Papers with LLM distillation
Teaching-Assistant-in-the-Loop: Improving Knowledge Distillation from Imperfect Teacher Models in Low-Budget Scenarios (2024.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated state-of-the-art (SOTA) performance across a wide spectrum of tasks. |
| Approach: | They propose a framework that leverages three signal types to improve efficiency within resource-constrained, imperfect teacher scenarios. |
| Outcome: | The proposed framework improves on four complex reasoning tasks by 20.79% compared to fine-tuning without any signals across datasets. |
DRP: Distilled Reasoning Pruning with Mathematical Skill-aware Step Decomposition for Efficient Large Reasoning Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing solutions to this problem are inference-time pruning and tuning-based distillation. |
| Approach: | They propose a framework that combines inference-time pruning with tuning-based distillation to enable efficient and accurate reasoning. |
| Outcome: | The proposed framework reduces token usage while improving accuracy on GSM8K and AIME tokens while avoiding performance drop. |
Multi-Granularity Semantic Revision for Large Language Model Distillation (2026.acl-long)
Copied to clipboard
Xiaoyu Liu, Yun Zhang, Wei Li, Simiao Li, Xudong Huang, Hanting Chen, Yehui Tang, Jie Hu, Zhiwei Xiong, Yunhe Wang
| Challenge: | Existing methods for generating large language models rely on student-generated outputs, which introduce generation errors and misguide the distillation process. |
| Approach: | They propose a multi-granularity semantic revision method for LLM distillation that corrects errors using teacher-generated tokens and re-generates the sequence to minimize errors. |
| Outcome: | The proposed method reduces errors and misguides distillation on student models and improves consistency between teacher and student outputs. |
Calibrated Progressive Distillation: Co-Designing Curriculum and Target Mixing for Knowledge Distillation of Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for knowledge distillation address the teacher–student capacity gap by mixing teacher and student distributions in the distillation target or using curriculum learning to sequence training from easy to hard examples. |
| Approach: | They propose a white-box KD framework that co-designs curriculum scheduling and target mixing through a unified difficulty-aware principle. |
| Outcome: | The proposed framework outperforms existing methods while reducing training runtime by over 10%. |